Model and estimate \(f(X)\), denoted \(\hat{f}(x)\)
Define metric to evaluate estimated model; e.g. \(\hat{MSE}(x) = E[(Y-\hat{f}(X))^2|X=x]\)
Supervised Learning
Give learner training data
Learner returns model \(\hat{f}(x)\)
Supervised Learning
Give test data estimated model \(\hat{f}(x)\)
Compare predicted outcomes from \(\hat{f}(x)\) with observed
Mean Squared Error Decomposition
Recall: \(MSE\) for estimate at \(X=x\) can be decomposed into \[MSE_{\hat{f}}(x)=E[(Y-\hat{f}(X))^2|X=x]=[f(x)-\hat{f}(x)]^2+Var(\epsilon)\]
Consider taking expectation marginally (i.e., across \(Y\) and \(X\)).
Can show \[E[(Y-\hat{f}(X))^2]=E_x[\text{bias}(\hat{f}(x))^2]+E_x[\text{Var}(\hat{f}(x))]+\text{Var}(\epsilon)\]
where \(\text{bias}(\hat{f}(x))=\text{E}[\hat{f}(x)]-f(x)\)
Bias-variance trade-off
Above means bias and variance of model increases expected model error
Creates tradeoff:
Higher complexity: decreased bias but increased variance
Lower complexity: increased bias but decreased variance
Bias-variance trade-off
Classification
Suppose instead response \(Y\) is categorical
e.g. cancer stage is one of \(C=(0, 1, 2, 3, 4)\) where \(0\) indicates cancer-free
Goals:
Build classifier \(\hat{f}(X)\) that maps a category from \(C\) to future observation \(X\)
Assess uncertainty in each classification
Understand roles of predictions \(X=(X_1, X_2, \ldots, X_p)\)
Classification
What to model?:
Let \(p_k(x)=\text{Pr}(Y=k|X=x)\), \(k=1,2,\ldots,K\)
Denoted as the conditional class probabilities at \(x\)
If these are known, can define classifier at \(x\) by
\(f(x)=j\) if \(p_j(x)=\text{max}[p_1(x), \ldots, p_K(x)]\)
Denoted as the Bayes optimal classifier at \(x\)
Classification metrics
Basic:
\(\text{accuracy}=\frac{\text{# correct predictions}}{\text{# test instances}}\)
\(\text{error}=1-\text{accuracy}\)
These are in general not sufficient (why?)
Confusion Matrix
Want to correctly identify positive and negative instances accurately
For positive instances, have the following metrics: \[\begin{align}
\color{green}{PPV=\frac{TP}{TP+FP}} && \color{red}{Sensitivity=\frac{TP}{TP+FN}} && F_1=\frac{2}{\frac{1}{PPV}+\frac{1}{Sensitvity}}
\end{align}\]
Confusion Matrix
Example
During the COVID-19 pandemic, different metrics to quantify risk:
K-nearest Neighbor (KNN)
Simple and flexible algorithm:
Distance Metrics
Different metrics can dramatically change neighborhoods:
